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Abstract — One  can  easily  envision  future  military  operations  and 
emerging  civilian  requirements  (e.g.  intelligent  unmanned  vehicles 
for  urban  warfare,  intelligent  manufacturing  plants)  that  will  be 
both  complex  and  stressing  and  will  demand  innovative  sensors 
and  sensor  configurations.  The  goal  of  our  research  into  Sensors 
as  Robots  is  to  develop  a  cost  effective  and  extendable  approach 
for  providing  surveillance  for  a  variety  of  applications  in 
dynamically  changing  military  and  civilian  environments.  Within 
Sensors  as  Robots,  we  foresee  a  new  sensor  archetype.  In  this 
paradigm,  sensors  and  algorithms  will  be  autonomously  altered 
depending  on  the  environment.  Radars  will  use  the  same  returns 
to  perform  detection  and  discrimination,  to  adjust  the  platform 
flight  path  and  change  mission  priorities.  The  sensors  will 
dynamically  and  automatically  change  waveform  parameters  to 
accomplish  these  goals.  Disparate  sensors  will  communicate  and 
share  data  and  instructions  in  real-time.  Intelligent  sensor  systems 
will  operate  within  and  between  sensor  platforms  such  that  the 
integration  of  multiple  sensor  data  provides  information  needed  to 
achieve  dynamic  goals  and  avoid  electromagnetic  fratricide. 
Intelligent  sensor  platforms  working  in  partnership  will  increase 
information  flow,  minimize  ambiguities,  and  dynamically  change 
multiple  sensors’  operations  based  upon  a  changing  environment. 
Concomitant  with  the  current  emphasis  on  more  flexible  defense 
structures,  Sensors  as  Robots  will  allow  the  appropriate 
incremental  application  of  remote  sensing  assets  by  matching 
resources  to  the  situation  at  hand. 

In  this  paper,  we  discuss  the  development  of  a  futuristic 
intelligence,  surveillance  and  reconnaissance  concept  utilizing  the 
innovative  integration  of  cutting  edge  technologies  such  as: 
knowledge-based  signal  processing,  robotics,  wireless  networking, 
waveform  diversity,  the  Semantic  Web,  advanced  computer 
architectures  and  supporting  software  languages.  This  concept  is 
projected  as  an  autonomous  constellation  of  air,  space,  and 
ground  vehicles  that  would  offer  a  robust  paradigm  to  build 
toward  future  deployments. 

I.  THE  CONFLUENCE  OF  FACTORS 

Currently,  many  national  defense  institutions  are  undertaking 
major  alterations  in  their  capabilities,  from  forces  designed 
during  the  Cold  War  to  those  adapted  to  21st  century 
adversaries  including  terrorism.  This  change  has  been  driven 
by  a  changing  threat  as  well  as  technology  innovations, 
especially  infonnation  technology.  As  the  military  services 
attempt  to  increase  the  agility  and  versatility  of  their  forces, 
they  also  see  a  need  to  increase  the  capabilities  of  military 
intelligence,  surveillance  and  reconnaissance  (ISR)  to  support 
the  new  weapon  systems  and  operating  methods  against  these 
new  threats.  Intelligence,  surveillance,  and  reconnaissance 
refer  to  efforts  to  collect  information  and  use  it  in  militarily 
significant  ways1. 

Based  on  changes  in  both  the  threat  and  in  available 
technology,  national  defense  organizations  either  are  in  the 
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midst  of,  or  recognize  the  need  to  “transform”.2 
Transformation  in  the  context  of  large  organizations  is 
generally  recognized  as  a  process  of  radical  change  involving 
technology,  organization,  and  concepts  of  employment. 
Another  expression  used  in  military  terminology  that  also 
expresses  the  idea  of  transformation  is  “revolution  in  military 
affairs”.  There  are  at  least  two  competing  perspectives  on 
what  constitutes  “transformation.”  To  some,  transformation  is 
characterized  as  a  discontinuous  or  leap-ahead  change.3  This 
view  supports  those  who  believe  it  is  necessary  to  move 
money,  manpower,  and  particularly  patterns  of  thinking 
(“doctrine”)  away  from  current  weapon  systems  and  methods, 
to  entirely  new  technologies  and  procedures.  They  perceive 
that  resources  are  being  wasted  on  the  older  systems,  and  the 
only  way  to  accomplish  change  is  to  do  so  in  a  radical  way. 
The  United  States  Navy’s  shift  in  the  1920s  and  30s  from  the 
battleship  to  the  aircraft  carrier  as  its  centerpiece  weapon 
system  could  be  considered  an  example  of  a  leap-ahead 
change,  even  as  the  battleship  remained  in  service  until  the 
1980s.  The  US  Army’s  plan  to  replace  its  tank  force  in  favor 
of  much  lighter  vehicles  and  other  technologies  may  be  seen 
as  an  attempt  to  achieve  similar  change.4 

Others  tend  to  classify  transformation  as  incremental  change 
using  current  technologies  in  new  ways,  with  an  end  result  of 
far-reaching  improvement  over  time.  They  express  concern 
that  the  future  is  unknown;  if  proven  capabilities  are 
eliminated  for  new  ones,  those  new  capabilities  may  not 
address  the  currently  unforeseen  threat  any  better  than  today’s 
technology.  A  proven  technology,  however,  may  be  able  to  be 
adjusted  to  meet  that  unknown  situation. 

Whatever  one’s  position  is,  we  believe  that  the  Sensors  as 
Robots  program  is  in  harmony  with  the  broad  idea  of 
transformation.  By  innovative  integration  of  cutting  edge 
technologies,  the  Sensors  as  Robots  program  will  be  focused 
on  developing  the  system-level  concepts  to  enable 
autonomous  and  agile  ISR  operations. 

II.  MOTIVATION 

A  Boy  Scout  troop  is  charged  with  cleaning  the  town  park. 
Initially,  each  boy  is  assigned  a  specific  task  and  timeline. 
However,  soon  into  the  exercise,  and  as  a  result  of  some 
minor  and  perhaps  major  situational  changes,  the  original 
individual  marching  orders  are  modified,  or  perhaps  even 
ignored,  on  the  fly.  In  any  event,  however,  each  member  of 
the  troop  does  his  part  to  accomplish  the  goal  of  cleaning  the 
park.  In  the  end,  the  mission  is  successfully  completed. 
Independent  entities  (the  Scouts)  have  operated  in  an 
autonomous  manner  using  cognitive  reasoning  in  responding 
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to  real-time  changes  in  the  environment,  to  accomplish  a  pre¬ 
assigned  mission.  The  individual  Scouts,  by  sensing  the 
behavior  and  activities  of  their  colleagues  in  response  to  the 
changing  environment,  and  by  communicating  the  necessary 
data  and  information,  achieved  a  collective  response  that 
resulted  in  a  “successful”  operation.  This  “system”  worked 
as  a  result  of  the  autonomous  and  intelligent  interaction 
between  the  individual  “subsystems”. 

Imagine  it  is  the  year  2013  and  friendly  sensor  assets  are 
hovering  over  the  enemy  fighter  aircraft  as  shown  in  figure  1 . 
This  scene  could  be  taking  place  today.  However  what  this 


Figure  1.  Futuristic  Scenario 

figure  does  not  show  is  that  each  sensor  is  performing 
independently  as  they  are  today  and  in  concert  with  all  other 
sensors,  similar  to  our  Boy  Scouts.  Each  sensor  is 
autonomous  trying  to  meet  their  independent  goals  (e.g. 
surveillance,  imaging,  tracking,  etc.)  but  are  working  in 
concert  providing  and  receiving  data  and  information  from 
near  and  distant  sensors  that  help  them  in  achieving  their  goals 
and  assisting  other  sensors  to  achieve  theirs.  Each  sensor 
system  will  have  intelligent  software  within  their  processing 
system  that  can  manage  its  resources,  communicate  with  other 
sensor  systems,  provide  data/information,  and  fulfill  requests 
from  other  sensors  and  human  operators. 

Consider  the  following  scenario  where  a  radar  is  going  to  drop 
track  due  to  an  obstruction  from  a  mountain.  Previous 
communications  and  sharing  of  data/infonnation  has  occurred 
such  that  this  radar  only  needs  to  send  a  small  encoded 
message  to  all  nearby  sensor  platfonns  in  advance  stating  that 
if  track  X  continues  in  its  present  direction  the  radar  will  lose 
the  target  because  of  an  obstruction.  Another  sensor  platform 
will  accept  the  primary  responsibility  for  tracking  the  target 
based  upon  set  criteria  established  within  their  intelligent 
system  software,  confirms  the  track  takeover,  and  reports  the 
results  to  the  user.  In  the  interim,  both  sensors  will  track  the 
target,  share  their  raw  data,  and  using  stereo  processing  and 
imaging,  they  will  pinpoint  exactly  where  the  target  is  located 
and  identify  it  with  minimum  error;  all  performed  without 
human  intervention. 

There  will  be  multiple  levels  of  intelligent  processing 
performed  throughout,  for  the  sharing  of  data  and  information 
between  sensors  on  manned  and  unmanned  platforms, 
between  platforms,  between  combat  areas,  and  between 
command  centers.  Missions  and  goals  will  change  in  real¬ 
time  at  all  levels  (i.e.  sensor,  platform,  combat  area,  etc.)  and 
sensors  will  be  tasked  to  change  accordingly,  whether  it  is  to 


detect  targets,  track  them,  help  identify  targets  with  other 
sensors,  perfonn  battle  damage  assessment,  or  guide  weapons 
to  a  target.  Sensors  are  resources  that  will  act  independently 
but  in  concert  as  global  goals  and  missions  change  in  real-time 
to  meet  requirements  in  a  changing  environment.  Placing 
intelligence  within  sensors  and  sensor  platforms  will  allow 
them  to  adapt  in  a  more  dynamic  and  cooperative  manner  just 
as  we  deploy  and  task  our  combat  forces.  To  meet  these  goals 
we  must  put  intelligence,  communications,  robustness,  and 
variability  (e.g.  waveform  diversity)  within  each  of  our  sensor 
systems,  and  leverage  a  variety  of  technologies.  Sensors  as 
Robots  is  a  goal  oriented  concept  that  is  in  stark  contrast  to 
“stove  pipe”  systems  currently  deployed.  We  need  a  concept, 
to  motivate  and  direct  our  research  and  development,  which  is 
larger  than  an  autonomous  vehicle  with  sensors  onboard. 
Intelligent  software  processing  is  required  at  all  stages  of 
signal,  data,  and  system  processing  from  the  filtering, 
detection,  tracking,  imaging,  and  identification  stages  to 
guiding  of  weapons  to  a  target,  battle  damage  assessment, 
communications,  command,  and  control. 

Contemporary  radar  sensors  and  systems  in  general  are  not 
optimally  integrated  within  the  platform  on  which  they  reside. 
They  do  not  adequately  share  nor  receive  information  from 
other  systems  on  the  same  platform,  or  between  platforms. 
Sensor  platforms,  for  the  most  part,  are  a  heterogeneous 
collection  of  disjointed  sensors.  However,  the  intelligent 
integration  of  multiple  sensor  data  sets  could  create  valuable 
information  for  improved  overall  mission  performance.  The 
user  community  is  data  rich  and  information  poor  when  it 
comes  to  raw  data  from  sensor  systems.  Consistent  with  the 
Sensors  as  Robots  theme,  it  will  be  important  to  have  sensor 
systems  automatically  share  data  and  information  so  that  their 
composite  result  provides  maximally  useful  information  to 
the  user. 

Historically,  radar  systems  have  been  built  given  a  fixed  set  of 
requirements.  As  a  result,  bounds  on  operational  flexibility 
have  been  set.  Imbedded  processing  algorithms  were  derived 
under  assumptions  that  may  not  always  be  valid.  As  a  result, 
these  algorithms  may  not  be  well  matched  to  modern-day 
needs.  Most  radar  sensor  systems  operate  stand-alone  and  do 
not  communicate  with  other  systems  except  for  handoff,  and 
generally  perfonn  single  mode  operations.  In  the  future, 
simultaneous  multi-mission  multi-mode  operation  will  be 
required. 

Current  fielded  radar  system  software  is  difficult  and  costly  to 
change  or  port  to  new  processors  since  it  is  often  computer 
platfonn  dependent.  However,  there  are  efforts  underway  to 
minimize  the  effort  of  porting  software  to  different 
architectures.  One  effort’s  approach  developed  a  compiler 
that  will  input  MATLAB  code  and  generate  efficient  low 
level  code  that  will  run  on  a  distributed  environment  of 
commercial  off  the  shelf  hardware  of  field  programmable  gate 
arrays  (FPGAs).  This  approach  evolved  from  research 
funded  by  the  Defense  Advanced  Research  Project  Agency 
(DARPA).  The  reader  is  referred  to 

http://www.ece.northwestern.edu/cpdc/Match/Match.html  for 

more  information.  Another  approach  (also  partially  funded 
by  DARPA)  is  being  pursued  by  a  consortium  called  the 
VSIP  Forum.  They  are  developing  a  Vector  Scalar  Image 
Processing  (VSIP)  Library  (VSIPL)  (www.vsipl.org).  The 
VSIPL  primitives  are  low  level  and  are  currently  written  in  C. 
The  Core  profile  contains  515  functions  and  Core  Lite 
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contains  125  of  these  same  functions.  Both  Mercury  and  Sky 
Computers,  two  builders  of  special  purpose  architectures  for 
signal  processing,  support  VISIPL  functions.  A  third 
approach  funded  by  the  US  Air  Force  and  US  Missile 
Defense  Agency  is  investigating  the  benefits  of  converting 
MATLAB  code  to  Java,  C,  C++,  and  to  VSIPL  primitives. 
(Unpublished  final  report  for  USAF  contract  F30602-99-C- 
0141).  Pursuing  these  and  other  approaches  will  help  meet 
the  demands  of  lowering  the  cost  of  software  development 
and  portability. 

Sensors  as  Robots  foresees  a  new  sensor  archetype.  In  this 
paradigm,  sensors  and  algorithms  will  be  autonomously 
changed  depending  on  the  environment.  Radars  will  use  the 
same  returns  to  perform  detection  and  discrimination,  and 
provide  data  to  alter  the  platfonn  flight  path  and  change 
mission  priorities.  The  sensors  will  dynamically  and 
automatically  change  wavefonn  parameters  to  accomplish 
these  goals.  Dissimilar  sensors  will  communicate  and  share 
data  and  instructions  in  real-time.  Intelligent  sensor  systems 
will  exist  within  and  between  sensor  platfonns  such  that  the 
integration  of  multiple  sensor  data  will  provide  information 
needed  to  achieve  dynamic  goals  and  avoid  electromagnetic 
fratricide.  Intelligent  sensor  platforms  working  together  will 
increase  information  flow,  minimize  ambiguities,  and  alter 
operations  based  upon  a  changing  environment.  Consistent 
with  the  current  emphasis  on  a  more  flexible  defense 
structure,  Sensors  as  Robots  will  enable  the  incremental 
application  of  remote  sensing  assets  by  matching  appropriate 
resources  to  the  problem. 

The  targeted  application  for  Sensors  as  Robots  is  the 
Unmanned  Airborne  Vehicle,  (UAV).  In  the  regime  of  UAV- 
based  remote  sensing,  we  envision  the  ability  to  detect  and 
exploit  observable  phenomena  in  the  “Long,  mid  and  near” 
range  by  mimicking  the  integrated  use  of  the  five  natural 
senses  plus  memory  and  exploiting  our  understanding  of 
human,  animal  and  insect  behavior  for  deployment  and 
operation.  Improved  sensor  signal  and  data  processing  will  be 
gained  from  knowledge  based  and  “a  priori”  information, 
multiple  processing  paradigms,  and  sensor  fusion.  Through 
the  use  of  all  available  sensor  and  control  data,  autonomous 
maneuvering  (locomotion,  displacement  and  “right” 
placement)  of  UAV  sensors  can  be  achieved.  UAV  based 
sensors  will  have  to  accomplish  difficult  tasks  in  dynamic 
environments.  However,  existing  robotic  systems  accomplish 
simple  tasks  that  are  not  scenario  level  functions  and  they 
generally  do  not  operate  in  concert  with  other  systems.  While 
some  multi-function  robotic  systems  exist,  (factory  “system 
of  systems”),  they  are  operating  in  carefully  controlled 
environments.  This  research  initiative  will  strive  to  meet 
future  UAV  sensor  requirements  to  perfonn  multi-level 
autonomous  functions  dynamically,  in  the  real  world. 

Future  military  and  civilian  requirements  will  be  stressing  and 
will  require  innovative  sensors  and  sensor  configurations. 
However,  many  on-going  and  promising  research  and 
development  investigations  increase  our  confidence  that 
maturing  technologies  will  foster  success.  Examples  which 
will  augment  Sensors  as  Robots  and  advances  in  UAVs 
include  the  development  of  knowledge  based  space  time 
adaptive  processing  (KBSTAP)3,6  a  dynamic  software 
architecture  for  the  filtering,  detection,  and  tracking  stages  of 
radar  processing,  using  the  “non-homogeneity  detector”; 
United  States  Geological  Service  (USGS)  “map”  data7; 


archival  radar  data;  as  well  as  off  board  sensor  data  to  select 
the  most  appropriate  space  time  adaptive  processing  (STAP) 
training  data  for  improvements  in  filtering,  detection,  track, 
identification  and  handoff;  as  well  as  waveform  selection  and 
flight  planning.  As  a  result  of  recent  and  current  research 
efforts,  well-grounded  and  validated  signal  processing 
algorithms8,9  abound.  To  deal  with  the  enonnous  quantity  of 
data  expected  in  the  future  system  architectures,  increased 
processor  speeds  will  be  needed.  On-going  industrial 
activities  are  resulting  in  processor  speeds  that  are  doubling 
every  18  months.  At  the  same  time,  the  object  oriented 
software  paradigm  is  helping  to  reduce  the  cost  of  software 
via  software  reuse.  The  next  generation  Internet  and  the  AI 
(Artificial  Intelligence)  communities  are  providing  new 
software  tools  and  models  as  well  that  should  prove  to  be 
useful  in  this  initiative. 

The  confluence  of  on-going  research  objectives  can  be  built 
upon  as  we  move  towards  a  future  “Sensors  as  Robots” 
configuration.  For  example  consider  the  Autonomous 
Navigation  and  Sensing  Experimental  Research  (ANSER) 
program  executed  by  BAE  Systems  Australia  and  the 
University  of  Sydney’s  Australian  Centre  for  Field  Robots. 
In  these  experiments,  the  goal  was  to  demonstrate 
decentralized,  multiple  UAV  tactical-picture  compilation  of 
ground  targets.  Communications  equipment  was  used  to 
permit  direct  UAV-to-UAV  WLAN  and  remotely  piloted 
flights  over  a  test  range  in  New  South  Wales  were  conducted. 
Building  on  the  tremendous  success  of  programs  such  as 
ANSER,  one  of  our  goals  is  to  eliminate  the  need  for  remote 
piloting  so  that  the  Sensor  as  Robots  function  with  only  initial 
instructions  and  the  ability  to  reconfigure  flight  paths 
autonomously  as  the  mission  unfolds.  Clearly  the  initial 
success  of  Remotely  Piloted  Vehicle  programs  are  essential  to 
the  incremental  progress  that  will  lead  to  a  true  “Sensors  as 
Robots”  manifestation. 

Eventually,  technology  will  pennit  the  deployment  of 
multiple  UAV  platfonns  in  perforating  dynamic  scenarios. 
Each  UAV  could  have  a  suite  of  heterogeneous  sensors  that 
can  operate  autonomously  or  in  concert  with  other  platforms. 
In  this  manner  we  can  deploy  the  correct  number  and  type  of 
UAVs  to  meet  the  requirements  and  goals  of  the  deployment, 
reduce  risk,  and  minimize  the  use  of  expensive  or  manned 
platfonns.  Sensors  as  Robots  technology  will  help  to 
accelerate  this  process  for  both  military  and  civilian 
applications. 

III.  AN  INTELLIGENT  SENSOR  SYSTEM 

In  order  for  a  system  of  sensors  platfonn  to  share  and  receive 
infonnation  from  multiple  sources,  it  must  be  able  to 
communicate  and  understand  the  infonnation.  A  potentially 
powerful  solution  for  the  exchange  of  information  between 
heterogeneous  sensors  is  for  each  sensor  to  publish 
infonnation  based  upon  an  accepted  and  understood  fonnat 
(i.e.  ontology).  In  this  manner  when  a  sensor  publishes  its 
track  data,  for  example,  other  sensors  receiving  this 
infonnation  will  be  able  to  interpret  its  contents  without 
ambiguity.  Achieving  this  will  require  that  certain  basics  be 
established.  There  must  be  available  an  accepted  method  of 
defining  the  Earth’s  geometry  such  that  the  location  of  every 
element  is  defined  within  the  same  coordinate  system  and  that 
each  element  is  time  synchronized  with  the  same  clock  and 
all  communications  are  time  stamped. 
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Each  transmission  of  information  between  sensors  must 
describe  its  time  and  coordinates.  In  addition,  if  it  is  sharing 
track  or  target  data,  it  must  specify  a  unique  identifier,  the 
sensor  platform’s  velocity,  pitch,  yaw,  and  roll  and  Meta  data 
describing  the  transmitted  raw  data  along  with 
encryption/decryption  keys.  The  unique  identifier  will  allow 
the  receiving  sensor  to  acquire,  within  its  resident  database 
management  system,  all  of  the  sender’s  radar  characteristics. 
Sensor  characteristics  include  such  things  as  nomenclature, 
power  output,  bandwidth,  frequency,  antenna  pattern,  pulse 
width,  pulse  repetition  frequency,  etc.,  as  well  as  platform 
characteristics  as  to  the  position  of  the  antenna  on  the 
platform,  number  of  elements,  the  pattern  of  the  elements,  the 
pointing  vector  of  the  sensor,  etc.  An  ontology  is  needed  for 
defining  these  data  and  numerous  rules  such  that  the 
information  published  by  any  sensor  can  be  understood 
correctly  by  the  receiving  sensor  to  perform  functions  such  as 
sensor  fusion,  track  correlation,  and  target  identification. 


IV.  COMPONENT  BUILDING  BLOCKS 

In  order  to  be  successful,  the  Sensors  as  Robots  program  must 
leverage,  and  in  some  cases,  stimulate  advances  in  an  array  of 
technological  regimes.  These  include: 

•  Sensor  systems  and  signal  processing  research 
activities  to  develop  knowledge-aided  sensor  signal 
processing  techniques  to  maximize  the  use  of  prior 
knowledge  in  radar  signal  &  data  processing  for 
improvements  in  detection,  track,  identification  & 
handoff;  state-of-the-art  facilities  such  as  the  US  Air 
Force  Research  Lab  Signal  Processing  Evaluation, 
Analysis,  and  Research  (SPEAR)  facility  for  the 
evaluation  of  signal  processing  algorithms;  and 
technology  and  software  for  integrating,  managing, 
and  sharing  infonnation  between  multiple  sensors  and 
information  resources. 


Sharing  infonnation  between  sensors  on  the  same  platform  is 
also  required,  especially  if  one  or  more  sensors  are  adaptively 
changing  waveform  parameters  to  meet  the  demands  of  a 
changing  environment.  Figure  2  depicts  a  notional  intelligent 
sensor  system.  Each  of  the  sensors  has  its  own  signal  and 
data  processing  functional  capability.  In  addition  to  this 
capability,  an  intelligent  processor  to  address  fusion  between 
sensors,  communication  between  sensors,  and  control  of  the 
sensors  has  been  added.  A  primary  goal  is  to  be  able  to  build 
this  processor  such  that  it  can  interface  with  any  sensor  and 
communicate  with  the  other  sensors  using  ontological 
descriptions  via  the  intelligent  platfonn  network.  The 
intelligent  network  will  be  able  to  coordinate  the 
communications  between  the  sensors  onboard  and  to  off 
platfonn  sensor  systems.  There  are  approaches  that  can  be 
exploited  to  build  this  system  by  using  fiber  optic  or  wire 
links  onboard  the  platform.  Radio  frequency  (RF)  links  using 
Bluetooth  or  802.11  technologies  can  be  utilized  for  linking 
these  sensors  onboard  the  platfonn.  Between  platfonns,  other 
technologies  may  be  implemented  such  as  mobile  Internet 
protocol  over  RF  communications  links.  The 
communications  issues  need  to  be  addressed  for  the  sharing 
of  information  and  for  minimizing  the  potential  of 
electromagnetic  (EM)  fratricide.  The  intelligent  platform 
should  determine,  for  example,  if  there  is  EM  interference 
(EMI)  potential  when  a  sensor’s  antenna’s  main  beam 
pointing  vector  or  PRF  is  changed,  and  may  thereby  cause 
interference  to  a  receiving  sensor.  Rather  than  have  each 
sensor  on  a  platfonn  operate  as  an  independent  system, 
platforms  would  be  designed  as  a  system  of  sensors  with 
multiple  goals  managed  by  an  intelligent  platfonn  network 
that  can  manage  the  dynamics  of  each  sensor  to  meet  the 
common  goal(s)  of  the  platfonn. 


Figure  2.  An  Intelligent  Sensor  System 


Language/modcl  development  for  numerical  and  non- 

numerical  processing  efforts  and  activities  will  be 
required  to  provide  the  capability  to  easily  define  the 
signal,  data,  and  logic  based  sensor-processing 
functions.  Requirements  will  include  the  capability  to 
exercise  this  language  on  conventional  processors  for 
emulation  and  evaluation  of  single  and  multiple  sensor 
sub-subsystems  where  their  respective  information  for 
all  sensors’  use  is  fused.  Research  and  development 
activity  will  be  required  to  develop  the  capability  to 
store  and  access  multiple  dynamic  and  stable  data  and 
information  sources  as  well  as  the  ability  to  model  the 
processing  within  and  between  multiple  sensors  and 
communications  systems. 

Artificial  intelligence  (AI)  -  robotics  programs  will  be 
critical  to  the  Sensors  as  Robots  program  by 
leveraging,  for  example,  the  work  of  the  World  Wide 
Web  Consortium  (W3C)  and  activities  such  as  the 
Defense  Advanced  Research  Project  Agency’s 
(DARPA)  Agent  Markup  Language  (DAML) 
program,  which  will  lead  to  the  next  generation 
Internet  or  the  Semantic  Web.  The  Semantic  Web 
will  allow  one  to  develop  Web  pages  that  are  written 
such  that  software  can  read  and  understand  their 
contents.  The  next  generation  Web  is  being  designed 
in  a  manner  similar  to  a  large  knowledge  base  such 
that  one  can  define  ontologies  for  different  domains  of 
interest  (e.g.,  radar  or  sensors).  The  concept  of 
ontology  is  exactly  what  is  needed  to  achieve  a  system 
of  sensors  that  operate  in  a  collaborative  fashion  and 
eventually,  to  have  sensor  platforms  operate 
autonomously  as  a  robotic  system.  In  order  to  operate 
cooperatively  they  must  be  able  to  communicate, 
share  data  and  information,  and  understand  each  other 
and  their  environment.  Leveraging  the  approach  and 
technology  of  the  W3C  will  allow  for  the 
development  of  ontology  for  sensors  thereby  having 
one  knowledge  base  that  can  be  understood  by  all  new 
knowledge  based  sensor  systems  added  to  the  overall 
domain  including  communications,  radar,  electro- 
optical,  infrared,  acoustic,  etc.  This  approach  will 
allow  multiple  sensors  on  one  platfonn  to  inference 
and  fuse  data  and  information  from  all  sensors  on 
board.  It  will  also  allow  for  this  platfonn  to  share  and 
fuse  data  and  information  between  sensors  on  multiple 
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platforms  located  nearby  or  at  a  distance  within  a 
command  center.  The  construction  of  ontologies  is  a 
present-day  activity.  They  can  easily  be  found  on  the 
Web  and  can  be  used  to  build  and  share  information 
within  the  community  and  domain  of  interest.  The 
Sensors  As  Robots  program  will  leverage  the  object 
oriented  feature  of  inheritance  and  reference  the 
resource  descriptive  framework  (RDF)  (i.e.  an 
instantiation  of  an  ontology)  of  those  ontologies  that 
already  exist  and  then  add  those  additional  facts  and 
rules  required  for  one’s  own  needs.  For  example,  if  an 
RDF  describing  facts  and  rules  for  a  transmitter,  a 
receiver,  and  an  antenna  already  exists,  and  the  facts 
and  rules  meet  ones  needs,  then  they  should  be 
referenced  in  the  radar  ontology  that  one  is  building  as 
opposed  to  building  anew.  Ultimately,  it  will  be 
possible  to  develop  the  AI  algorithms  and  processing 
rules  within  various  sensor  types  such  as  radar, 
electro-optical,  acoustic,  infrared  and  perform 
multiple  tasks  such  as  imaging,  detection,  tracking, 
target  identification  and  to  develop  ontologies  for 
multiple  sensor  types  and  communication  systems  so 
that  they  can  exchange  data  and  information  in  a 
coherent  and  timely  fashion. 

Waveform  diversity  encompasses  both  the  adaptation 
of  waveforms  in  order  to  compensate  for  the 
environment  and  multi-user  interference,  as  well  as 
receiver  structures  that  employ  a  priori  and  estimated 
knowledge  to  separate  desired  information  from 
multiple  users  and  jamming.  Effort  will  be  needed  to 
investigate  waveform  diversity  techniques  that 
enhance  the  performance  of  multiple  sensor  types. 
Research  efforts  to  develop  algorithms  and  procedures 
to  alleviate  fratricide  issues  related  to  dynamic 
frequency  assignment  in  and  between  moving  and 
fixed  platfonns  must  be  undertaken.  Development  of 
communication  protocols  and  techniques  to  maintain 
communications  while  dynamically  changing 
waveform  parameters  will  be  needed. 

Communication  models  and  procedures  to  develop 
fault  tolerant  techniques  for  the  multiple  sensors  and 
communication  devices  to  communicate  within  a 
platform  will  be  required.  Similarly,  fault  tolerant 
methodologies  for  multiple  platforms  to  communicate 
by  leveraging  different  commercial  protocols  such  as 
Bluetooth,  802.11,  and  mobile  IP  are  needed.  These 
technologies  will  be  the  bond  that  connects  supporting 
technologies  such  as  operations  planning,  mission 
planning,  reasoning,  decision  making  and  distributed 
real-time  computing  and  control.  The  development  of 
wideband  communications  technologies  to  be  used 
between  platforms  to  transmit  and  receive  large 
amounts  of  data  such  as  data  cubes  and  their  proper 
ontological  header  data,  in  real  time  will  be  essential. 

Software  development  activities  to  leverage  languages 
such  as  MATLAB  and  the  results  of  language/model 
development  efforts  to  create  a  library  of  algorithms 
will  be  needed.  Efforts  to  develop  techniques  and 
tools  to  automatically  map  these  language 
characterizations  to  different  computer  architectures 
(and  their  languages,  Java,  VSIP,  C++)  must  be 
initiated.  On-going  activities  to  develop  methods  to 


minimize  the  cost  to  maintain  and  upgrade  software 
on  fielded  systems  will  be  watched  closely  and  used 
as  needed. 

•  Computer  architectures  research  efforts  to  develop 
fault  tolerant  computer  architectures  that  can  process 
signal,  data,  and  logic/control  of  a  real  time  sensor 
system  are  needed.  Effort  to  develop  architectures 
able  to  accommodate  both  numerical  and  non- 
numerical  processing  and  be  able  to  store  and  retrieve 
large  amounts  of  data  describing  such  entities  as 
ground  truth,  map  data,  intelligence  data,  battle 
damage  assessments  will  be  required.  Programs  to 
acquire  architectures  able  to  process  multiple  tasks  in 
parallel  with  the  same  sensor  data  (e.g.  tracking  and 
imaging),  and  be  able  to  spawn  off  multiple  task 
instantiations  of  each  with  different  algorithms  for 
comparisons  in  real  time  as  the  environment  changes 
must  be  started.  In  addition,  architectures  able  to 
receive  data,  information,  and  control  from  multiple 
on  board  and  off  board  sources  such  as 
communications  links  and  other  sensors  will  be 
essential. 

Other  technological  advances  such  as  robotic  locomotion, 
wireless  networking,  detection  and  classification  algorithms 
for  multi-modal  inverse  problems,  etc.  will  be  closely 
monitored  as  well,  and  where  appropriate,  leveraged  into  the 
overall  Sensors  as  Robots  program. 

V.  INITIAL  DIRECTION 

The  development  of  the  Sensors  as  Robots  concept  will 
proceed  using  a  top  down  systems  level  methodology  in 
which  advances  in  the  COMPONENT  BUILDING  BLOCKS 
above  are  utilized  and  integrated.  For  example  we  would 
incorporate  embedded  AI  and  software  advancements, 
leverage  Semantic  Web  technologies  (i.e.  XML, 
ontologies,...),  employ  intelligent  sensor  and  platform 
management  algorithms,  capitalize  on  ultra  wideband,  optical, 
and  wireless  advancements  (e.g.  mobile  IP,  802.11, 
Bluetooth),  and  use  human  engineering  interface  control  for 
intelligent  sensor  systems. 

The  system  strategy  that  will  be  employed  in  the  Sensors  As 
Robots  program  is  basically  a  three-phase  approach  (Figure 
3):  (1)  development  of  a  baseline  Sensors  As  Robots 
prototype  utilizing  state-of-the  art  Component  Building 
Blocks  (Version  1.0);  (2)  Periodic  Upgrade  and  integration  of 
advancements  in  Component  Building  Blocks  (Version  1.1, 
Version  1.2,  etc.);  (3)  Release  of  an  approved  version  to  the 
appropriate  sponsor  for  implementation.  This  phased 
approach  will  be  iterated  in  a  spiral  fashion  as  knowledge, 
technologies,  and  approaches  mature. 


Figure  3.  Sensors  as  Robots  Methodology 

An  objective  of  the  Sensors  as  Robots  approach  is  to 
accelerate  and  facilitate  application  of  mature  advanced 
technologies  (the  Component  Building  Blocks)  to  assist  in  the 
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development  of  a  cognitive  autonomous  sensor  system  and 
thereby  provide  new  operational  capabilities  that  will  make  a 
difference  in  military  and  civilian  applications.  At  this  point, 
the  process  by  which  a  Sensors  as  Robots  candidate  moves 
from  the  initial  phase  to  a  validated  model  is  very  flexible  but 
might  typically  be  described  as  follows. 

The  baseline  form  of  a  Sensors  as  Robots  implementation  will 
be  represented  by  a  collection  of  technology  programs  that 
are  combined  and  integrated  into  a  demonstration  carried  out 
in  the  laboratory  to  develop  or  enhance  a  military  capability 
or  a  needed  civilian  application.  Generically,  this  implies 
identifying  significant  operational  shortfalls,  matching  them 
up  with  technology  programs  ready  to  focus  on  important 
applications,  and  responding  to  a  user-sponsor  who  believes 
that  application  is  important  to  his  mission.  The  result  of  a 
phase  1  effort  will  be  a  Sensors  as  Robots  system  comprised 
of  the  most  mature  technology  building  blocks  available  and 
will  be  labeled  Version  1.0.  It  will,  however,  be  just  a  starting 
point.  Version  1.0  will  embody  a  rudimentary  autonomous 
system;  however,  it  will  serve  as  a  demonstration  platfonn  as 
we  build  towards  a  more  sophisticated  system. 

As  technology  advances  are  achieved  in  the  Component 
Building  Blocks,  judicious  selection  and  integration  of  these 
advances  will  be  inserted  during  phase  2  and  the  Sensors  as 
Robots  model  upgraded  to  Version  1.1,  1.2,  etc.  During  this 
phase,  each  version  will  meet  certain  criteria  with  respect  to 
interoperability,  functional  performance,  robustness,  etc.  At 
some  stage,  when  a  certain  level  of  specified  perfonnance  is 
achieved,  the  Sensors  as  Robots  model  is  released  as  Version 
2.0.  Of  course  this  model  must  be  thoroughly  coordinated 
with  potential  sponsors  throughout  its  development  phases. 

In  this  manner,  and  as  the  methodology  is  refined,  the  Sensors 
as  Robots  program  will  provide  a  mechanism  for  achieving  a 
cognitive,  autonomous  sensor  system. 

VI.  SUMMARY 

In  future  scenarios,  unattended  air  and  ground  based  sensors 
will  be  called  upon  to  accomplish  difficult  tasks  in  dynamic 
environments.  For  example,  we  foresee  military  applications 
in  the  vein  of  the  surveillance  of  hostile  urban  battlefield 
situations  as  well  as  monitoring  functions  inherent  to 
peacekeeping  activities;  civilian  applications  such  as  crop 
production  assessment/management  and  traffic 
assessment/management;  industrial  applications  that  include 
factory  smoke  emissions  monitoring,  production  assembly 
assessment  and  management,  facility  management,  and 
inventory  shipment  and  tracking. 

However,  existing  UAVs  and  mobile  robotic  systems  can 
only  accomplish  straightforward  or  repetitive  tasks  which  are 
not  scenario  level  functions  and  they  generally  do  not  operate 
in  concert  with  other  systems.  More  will  be  needed  in  the 
future.  Propelled  by  the  twin  factors  of  transfonnation  in 
military  affairs  and  the  dramatic  increase  in  technological 
innovation,  we  believe  that  the  environment  is  right  for  the 
introduction  of  a  new  concept  in  sensor  system  technology. 
This  concept  would  embody  cognitive,  autonomous  sensor 
system  operation  where  the  types  and  numbers  of  sensors 
would  be  matched  to  the  task  at  hand.  Sensors  will 
collaborate  with  each  other  by  sharing  information,  sensing 
the  environment,  adapting  operation  as  necessary,  and  will  do 


this  by  incorporating  artificial  intelligence,  capitalizing  on 
advancements  in  the  Semantic  Web,  employing  intelligent 
sensor  &  platform  management  algorithms,  exploiting  ultra¬ 
wide  band  communication,  optical,  &  wireless  advancements 
and  by  utilizing  human  engineering  interface  control  for 
intelligent  sensor  systems. 

The  Sensors  as  Robots  initiative  is  a  forward  looking 
initiative  that  we  believe  will  act  as  a  catalyst  and  vehicle  for 
stimulating  the  developments  needed  to  make  such  future 
sensor  systems  possible. 

VII.  APPENDIX  A:  ROBOTIC  FUNDAMENTALS10 

The  word  robot  was  derived  from  the  Czech  word 
for  forced  labor  by  Karal  Capek  in  R.U.R  (Rossum’s 
Universal  Robots),  published  in  1921.  The  modem  definition 
of  a  robot  classifies  it  as  a  machine  with  these  features:  an 
algorithm  for  instruction,  moving  parts  for  action,  sensors  to 
discern  items  of  interest  from  everything  else,  a  process  to 
select  appropriate  actions,  and  a  controller  to  govern  overall 
operations. 

A  major  focus  of  the  robotics  industry  is 
manipulation,  i.e.  the  arm  and  hand  action  required  for 
automation  of  the  manufacturing  process.  Most  important 
among  these  are  the  revolute  and  prismatic  joint,  the 
rectangular,  cylindrical,  spherical,  and  articulated  ann.  The 
industrial  robot  hand,  multi-fingered  with  force-sensing  wrist, 
compliments  the  joints  and  arms  described  above.  Mobile 
ground  based  robotic  systems  require  legs  and  feet,  or 
wheels/ tracks.  Airborne  robots  and  underwater  equivalents 
require  more  sophisticated  mechanisms  for  locomotion.  A 
mobile  robot  must  also  have  a  navigation  system.  In  primitive 
short  range  robotic  systems,  dead  reckoning  is  adequate. 
Today,  electronic  signals  such  as  GPS  (global  positioning 
system)  aid  in  long  range  navigation.  This  is  especially 
important  in  air  and  space  craft. 

Robots  may  incorporate  sensors  that  imitate  the 
senses.  In  the  animal  kingdom,  stereo  vision  and  the  visual 
cortex  enable  image  fonnation  and  analysis.  In  robots, 
passive  and  active  electromagnetic  sensors  may  function  in 
this  capacity,  albeit  not  to  the  level  of  human  expectations. 
Force  and  torque  sensors  are  routinely  implemented  using 
strain  gauges  in  robotic  systems.  Acoustic,  scent,  and  heat 
sensors  are  also  readily  available.  Integration,  as  in  the 
human  brain,  requires  work  on  several  problems 
simultaneously,  akin  to  parallel  processing.  Humans  also 
think  on  several  levels  at  once,  thus  establishing  a  hierarchy. 
This  is  also  of  interest  in  robotic  research. 

Teaching  a  robot  is  also  a  challenge.  Among  the  simplest  to 
teach  are  fixed  stop  robots  used  for  pick-and-place  operation. 
In  servo-controlled  robots,  learning  may  be  more  difficult.  A 
servomechanism  is  a  device  that  knows  where  a  machine  is 
and  should  be  in  its  environment.  This  permits  the  controller 
to  compare  where  the  machine  should  be  and  how  to  get  it 
there.  There  are  many  ways  to  teach  a  servo-controlled  robot. 
In  one  approach,  a  human  can  guide  a  robot  through  its 
various  motions,  recording  position  and  time  information. 
Alternatively,  robots  may  be  programmed  using  computer 
languages  specially  designed  to  describe  motion  and  angles. 
One  popular  industrial  method  combines  these  techniques. 
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Intelligent  teleoperated  systems  may  have  limited 
computerized  control.  With  supervision,  an  onboard  computer 
may  makes  decisions,  or  an  operator  can  take  control  of  the 
process. 

All  of  this  begs  an  important  question,  “Why  aren’t  there 
more  robots  in  the  work  place?”  Two  problems  impede 
progress.  Robots  are  not  accurate  enough  and  need  greater 
intelligence.  When  the  human  brain  is  compared  to 
computers,  it  is  the  more  complex  among  any  man  made 
device.  Symbolic  processing  is  a  small  part  of  how  the  brain 
works.  Humans  also  have  associative  memory  and  the  brain 
stores  information  in  such  a  way  that  limited  words  or  images 
can  evoke  memory  of  a  complicated  series  of  events.  Robots 
will  require  this  level  of  sophistication  in  future  work  place 
environments. 

Humans  understand  structure,  can  plan  to  solve  a  problem, 
and  then  carry  it  out,  adapting  along  the  way.  The  brain 
doesn’t  store  information  about  simple  tasks  or  other 
knowledge  all  by  itself.  Information  is  stored  within  the 
context  of  events  and  ideas.  Also,  a  person  can  apply  the 
knowledge  learned  about  one  situation  to  another.  This  type  of 
learning  is  called  generalization.  Even  the  simplest  task 
involving  generalization  may  be  well  beyond  the  capability  of 
modern  computers  or  robotic  systems.  As  a  result,  we  use 
expert  systems  to  mimic  human  behavior,  in  order  to  facilitate 
the  robotic  process. 

In  computers,  expert  systems  are  programs  that  follow  a  menu 
or  recipe.  Each  one  contains  a  body  of  information,  a  method 
of  collecting  specific  data  concerning  a  problem  under 
consideration,  a  set  of  rules  to  process  the  data,  and  a  set  of 
instructions  for  taking  action. 

A  domain  expert  has  two  types  of  knowledge.  One  type  is 
data  about  the  problem  of  interest.  The  other  focuses  on 
method  of  using  that  data.  A  general  description  of  the 
problem  and  data  is  inadequate.  The  expert  system  must 
include  operations  that  may  be  perfonned  automatically  on 
the  data,  and  the  ability  to  reason  and  adapt  to  unexpected 
conditions.  Reasoning  includes  the  ability  to  translate 
experience  into  rules.  This  is  shallow  or  heuristic  reasoning. 
Deep  reasoning  is  concerned  with  analyzing  a  problem  to 
discover  its  basic  structure.  Both  kinds  are  used  by  experts. 

Significant  research  currently  focuses  on  teleoperations  and 
sensor  improvements.  Sensors  as  Robots  will  require  the 
transmission  of  more  realistic  information  to  their  human 
colleagues  -  what  is  called  telepresence.  This  may  require 
artificially  intelligent  robots.  An  intelligent  robot  still  needs  a 
plan  to  follow.  The  plan  should  include  intermediate  goals.  It 
should  also  be  able  to  deal  with  failures  and  -  most  of  all  -  the 
unexpected.  The  robot  also  has  to  learn  from  its  experiences. 
And  it  has  to  be  aware  of  what  it  knows  and  doesn’t  know. 

The  most  important  skill  an  intelligent  robot  will  need  is 
introspection,  the  ability  contemplate  its  method  of  reasoning. 
Introspection  will  enable  computers  to  distinguish  between 
what  it  knows  and  what  it  needs  to  learn.  In  a  dynamically 
changing  environment,  this  will  permit  robots  to  decide 
whether  to  act  or  to  pursue  additional  infonnation. 
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